PCT 



WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




Dlt 



INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 
G06F 3/00, 15/16, 17/30 



Al 



(11) International Publication Number: WO 98/06024 

(43) International Publication Date: 12 February 1998 (12.02.98) 



(21) International Application Number: PCT/US97/ 1 2848 

(22) International Filing Date: 31. July 1997 (31.07.97) 



(30) Priority Data: 
08/691.852 



I August 1996 (01.08.96) 



US 



(71) Applicant: IMMERSION HUMAN INTERFACE CORPORA- 

TION [US/US1; 2158 Paragon Drive. San Jose. CA 9513! 
(US). 

(72) Inventors: ROSENBERG. Louis, B.; 849 Palomino Drive, 

Pleasanton, CA 94566 (US). TAN. Sian. W.; 1971 West 
Middlefield Road #2, Mountain View, CA 94043 (US). 

(74) Agents: RIEGEL, James. R. et al.; Hickman Beyer & Weaver, 
P.O. Box 61059. Palo Alto. CA 94306 (US). 



(81) Designated States: AU. CA, CN. JP. MX, RU, European 
patent (AT, BE, CH. DE, DK. ES. FL FR. GB, GR. IE. 
IT. LU. MC. NL, PT. SE). 



Published 

With international search report. 

Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 
amendments. 



(54) Title: METHOD AND APPARATUS FOR PROVIDING FORCE FEEDBACK OVER A COMPUTER NETWORK 



(57) Abstract 

An internet network force feedback system (10) which 
includes a server machine (18). a client machine (14) provided 
with the force feedback device (24), and one or more additional 
client machines (16) each of which may be provided with 
additional force feedback devices (26). Tne server machine 
is a computer or processor running TCP/IP server software and 
is connected to the Internet Client machine (14) runs internet 
browser software. The force feedback device (24) has sensors 
and actuators to monitor the input of the user and transmit the 
input to the client and receive response from the client and 
provide the force feedback sensations to the user. 
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Method and Apparatus for Providing Force Feedback Over a Computer Network 

Background of the Invention 

This invention relates generally to human/computer interfaces, and more particularly to 
human/computer interfaces with force feedback that can operate over a network. 

10 The Internet has, of late, become extremely popular. The origins of the Internet date back 

several decades to a U.S. government sponsored military/research/business wide area network 
(WAN) that was designed to remain operational even in the event of the catastrophe, e.g. a major 
earthquake or a nuclear war. To accomplish this goal, robust protocols and systems were 
developed which allowed a geographically distributed collection of computer systems to be 

15 connected as a WAN such that the loss of a particular computer, or group of computers, would 
not preclude the continued communication among the remaining computers. 

While the use of the Internet has been prevalent for many years now, its use has been 
limited by the arcane and difficult commands required to access the various computers on the 
network. To address this problem, a protocol known as the "World Wide Web" or "WWW" was 
20 developed to provide an easier and more user-friendly interface for the Internet. With the World 
Wide Web, an entity having a domain name creates a "web page" or "page" which can provide 
information and. to a limited degree, some interactivity. 

A computer user can "browse", i.e. navigate around, the WWW by utilizing a suitable 
web browser and a network gateway (e.g., an Internet Service Provider (ISP)). For example, 
25 UUNET, America Online, and Global Village all provide Internet access. Currently, the most 
popular web browser, known as the Netscape® Navigator®, is made by Netscape Corporation of 
Mountain View. California. The web browser allows a user to specify or search for a web page 
on the WWW, and then retrieves and displays web pages on the user's computer screen. 

The Internet is based upon a transmission protocol known as "Transmission Control 
30 Protocol/Internet Protocol" (or 'TCP/IP" for short), which sends "packets" of data between a 
host machine, e.g. a server computer on the Internet, and a client machine, e.g. a user's personal 
computer connected to the Internet. The WWW is an Internet interface protocol which is 
supported by the same TCP/IP transmission protocol. Intranets are private networks based upon 
Internet standards, and have become quite common for managing information and 
35 communications within an organization. Intranets, since they adhere to Internet standards, can 
often use the same web browser software and web server software as are used on the Internet. 

A web page typically includes static images and text. The images and text are specified 
in a "HyperText Mark-up Language" ("HTML") file that is sent from the web server to the client 
machine. This HTML file is parsed by the web browser in order to display the text and images 
40 on the display of the client machine. Other standardized languages or protocols are also being 
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5 developed for use with the Internet and the World Wide Web. For example, the Virtual Reality 
Modeling Language (VRML) is used to provide visual virtual 3-D environments and allow one or 
many users to navigate through and interact as "avatars" in such an environment using a client 
computer system. 

The Internet and the WWW also permit sound data to be transmitted over the Internet. 
10 For example, references to sound files can be embedded in HTML pages and can be played by 
the web browser. Data "packets" coded in TCP/IP format can also be sent from one client 
machine to another over the Internet to transmit sound data. This last-mentioned technique forms 
the basis for Internet telephony. 

While the transmission of visual images (both static and dynamic, i.e. video), text, and 
15 sound over the Internet is well-known, the transmission of other types of sensory data has not 
been well explored. In particular, the transmission of data over the Internet pertaining to the 
sense of touch and/or force has not been established. "Force feedback" allows a user to 
experience or "feel" tactile sensations as provided through computational information. Using 
computer-controlled actuators and sensors on a force feedback device, a variety of realistic 
20 sensations can be modeled and experienced by the user. This useful and highly immersive 
sensory modality for interacting with the Internet has hereto been unavailable. 
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Summary of the Invention 



The present invention is related to the transmission of information pertaining to a subset of 
the sense of touch, i.e. the transmission of forces to a user over a computer network system. The 
"force feedback" provided by the methods and apparatus of the present invention enhance the 
10 sensory experience of the user to provide a richer, more interesting, and more enjoyable experience. 

In a preferred method of the present invention for providing force feedback over a network, a 
connection is first established between a server machine and a client machine. The client machine 
(typically a personal computer), has a visual display (e.g., a computer video monitor) and a force 
feedback human/computer interface device. Examples of force feedback human/computer interfaces 

15 (hereafter "force feedback devices") include force feedback joysticks, mice, trackballs, steering 
wheels, and yokes. In some embodiments of the present invention, the force feedback device 
preferably constrains movement to two degrees of freedom to match a two-dimensional configuration 
of the visual display. Three or more degrees of freedom of movement can be provided in other 
embodiments. Next, the client machine receives from the server machine both screen display 

20 information and force feedback information that is related to the screen display information. 
Preferably, the screen display information and force feedback information are encoded in an HTML 
web page file. Files and data in other protocols or languages can also be used, such as VRML. 
Next, the client machine displays on the monitor an image generated from the screen display 
information. The force feedback device provides a pointer event and/or a button event to the client 

25 with respect to the screen image, and a force feedback signal is computed based upon the detected 
events and the force feedback information stored on the client machine. Finally, a force feedback is 
provided to a user by the force feedback device based upon the force feedback signal. 

Preferably, the force feedback device is provided with a local microprocessor which 
communicates with the client machine. The force feedback device further includes sensors and 
30 actuators coupled to the local microprocessor such that the force feedback signal can take the form of 
a relatively high-level force feedback command. The local microprocessor parses the force feedback 
command to control the actuators of the human/computer interface in a control loop with the sensors. 

Another preferred method of the present invention for providing force feedback over a 
network establishes a connection between a first computer and a second computer over a network. 
35 The first computer includes a computer input device which develops a first computer input, and a 
second computer includes a visual display and a force feedback device for providing a second 
computer input. The computer input device of the first computer may or may not also be a force 
feedback device. A screen image is displayed on the monitor of the second computer that is 
associated with stored force feedback information. A computer input is received from the first 
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5 computer oyer the network, and a force feedback signal, based on the stored force feedback 
information and upon at least one of the first computer input and the second computer input, is 
determined. Finally, a force feedback is provided with the force feedback device in response to the 
force feedback signal such that it is correlated with the screen image on the second computer. 

A preferred method of the present invention for providing force feedback over a network 
10 supporting TCP/IP protocols includes: (a) sending a connection request from a client computer over 
a network supporting TCP/IP protocols to a web server connected to the network that is hosting a 
desired URL; (b) receiving, parsing, and interpreting {i.e. "processing") an HTML file at the client 
computer that was sent from the web server in response to the connection request, wherein the 
processing includes parsing an embedded force object reference having associated parameters and 
15 building a force object that is specified by the force object reference and the parameters; (c) 
developing a force feedback signal from the force object; and (d) providing force feedback to a user 
of the force feedback device coupled to the client computer in response to the force feedback signal. 
Preferably, the HTML file is a web page of the web server and the parsing is provided by an 
enhanced web browser. Even more preferably, the web browser is provided with a software "plug- 
20 in" to aid in the processing of the embedded force object reference, the building of the force object, 
and the development of the force feedback signal. 

A network force feedback system in accordance with the present invention includes a 
network, a first computer coupled to the network, and a second computer coupled to the network, 
where the second computer includes a visual display and a force feedback device capable of 

25 providing a second computer input to the second computer, and further being capable of providing 
force feedback to a user in response to a force feedback signal provided by the second computer. 
Preferably, the force feedback device includes a local microprocessor that communicates with the 
second computer such that the force feedback signal can take the form of a relatively high-level force 
command. The second computer develops an image on the visual display that is correlated to stored 

30 feedback information, such that the second computer produces the force feedback signal in response 
to at least one of information provided from the first computer and of the second computer input. 
The first computer can either be a server computer or, like the second computer, another client 
computer coupled to the network. The present invention therefore permits a computer user to 
experience force feedback over the network. 

35 In the network force feedback system, information pertaining to force feedback can be 

downloaded from a server computer to the second or "client" machine. Alternatively, force feedback 
signals can be sent over the network between a first "client" machine and a second "client" machine 
on a peer-to-peer basis, or from a server to one or more peers in a server-to-peer basis. 

The present invention adds a new sensory modality when interacting with a networked 
40 computer system. More particularly, force information can be either downloaded to a client machine 
from a server machine connected to the network, or force information can be passed between two or 
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5 more client machines on the network. When force information is downloaded from a server, the 
client machine parses and interprets the force information and directly controls its interaction with the 
force feedback device on an essentially real-time basis. In contrast, peer-to-peer or server-to-peer 
direct interaction over the network may be subject to some transmission ("latency") delays, but 
permits remote interactivity with a client's force feedback device. 

10 These and other advantages of the present invention will become apparent upon reading the 

following detailed descriptions and studying the various figures of the drawings. 
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Brief Description of the Ppu/j ngs 



Fig. 1 is a pictorial representation of the Internet, a web server machine, and two client 
machines; 

10 Fig- 2 is a block-diagram of a client machine used in the present invention; 

Fig. 3 is a block-diagram of a force feedback system in accordance with the present 
invention; 

Fig. 4a is a perspective view of a preferred human/computer interface ("force feedback 
device") of the present invention; 

1 5 Fig. 4b is a cross-sectional view taken along line 4b-4b of Fig. 4a; 

Fig. 5a is a perspective view of another preferred embodiment for a force feedback device in 
accordance with the present invention; 

Fig. 5b is a perspective view of a first alternate embodiment for the force feedback device of 

Fig. 5a; 

20 Fig. 5c is a perspective view of a second alternate embodiment of the force feedback device 

of Fig. 5a; 

Fig. 6 is a block diagram of a wide area network (WAN) based upon Internet TCP/IP 
protocol and supporting World Wide Web (WWW) HTML protocols in accordance with the present 
invention; 

25 Fig. 7a is flow-diagram of a "Acquire URL" process in accordance with the present 

invention; 

Fig. 7b is an example of an HTML file of the present invention sent from a web server 
machine to a client machine; 

Fig. 8 is a flow-diagram of the "Parse and Interpret HTML Component" step of Fig. 7a; 
30 Fig. 9 is a flow-diagram of the "Plug-in Interpret .IFF File" step of Fig. 8; 

Fig. 9a is an illustration of an image displayed on a visual display of a client computer as 
generated from a downloaded HTML web page file; and 
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Fig. 10 is a flow-diagram of a process for monitoring the "pointer state" of the force feedback 
device of the present invention. 
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5 Detailed Description of the Preferred FmhnHi^Pnfc 



In Fig. 1, a network system 10 includes a wide area network (WAN) such as the Internet 12, 
and a number of computers or "machines" coupled to the Internet 12. For example, a first client 
machine 14. a second client machine 16. and a web server machine 18, are coupled to the Internet 

10 12. 

As noted previously, both the Internet 12 and Intranets operate using the same TCP/IP 
protocols. This allows Intranets to use similar or the same server machine software and client 
machine software as are used in Internet 1 2 applications. Therefore, it will be apparent to those 
skilled in the art that the following descriptions apply equally well to Internet, Intranet, and other 
15 forms of network systems that are compatible with the processes and apparatus disclosed herein. 

The Internet 12 includes a number of nodes 20 that are interconnected by data transmission 
media 22. These nodes are typically routers, switches, and other intelligent data transmission 
apparatus which route "packets" of TCP/IP information to the desired destination. In some 
instances, the nodes 20 comprise an Internet service provider (ISP) 20a which allows a client 
20 machine to access the "backbone" of the Internet. Alternatively, client machines and web servers can 
be coupled directly into the backbone of the Internet. 

As noted previously, the present invention is directed to the implementation of force feedback 
over a network, such as the Internet 12. To provide a user of a client machine with the experience of 
force feedback, force feedback human/computer interfaces (hereafter "force feedback devices") 24 
25 and 26 can be provided as part of the client machines 14 and 16, respectively. The client machines 
14 and 16 are typically provided with computer video monitors 28 and 30 (which is one example of a 
"visual display"), respectively, which can display images II and 12, respectively. Preferably, forces 
developed by force feedback devices 24 and 26 are correlated with the images II and 12 of the client 
machines 14 and 16, respectively. 

30 The machines 14-18 are considered, in the language of the Internet, to be "resources," and 

each has its own unique Uniform Resource Locator or "URL." In one embodiment of the present 
invention, a client machine, such as client machine 14 or 16, sends a request for a "web page" 
residing on, for example, web server machine 18. This is accomplished by the client machine 
sending a connection request and a URL which specifies the address of the web page to the web 

35 server machine 18. The web server machine 18 then sends a web page 32 written in HTML format 
back to the requesting client machine where it is "cached" in the memory (typically the RAM, hard 
disk, or a combination of the two) of the client machine. In this embodiment of the invention, the 
image on the video display of the client machine is generated from the HTML web page file cached 
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5 on the client machine, and force feedback is provided to a user through the force feedback device as 
he manipulates a user manipulate object of the force feedback device. 

In another aspect of the present invention, a first client machine, such as client machine 14, 
and a second client machine, such as client machine 16, directly communicate force feedback 
commands to each other in standard TCP/IP protocol over the Internet 12. More particularly, client 

10 machine 14 can send force feedback and other information to the URL of the client machine 16, and 
the client machine 16 can send force feedback and other information in standard TCP/IP packets to 
the URL of the client machine 14. In this way, users of client machine 14 and client machine 16 can 
interact physically over the Internet 12. Of course, a server machine 18 can likewise directly 
communicate force feedback commands to a client machine 12 or 14, or all three machines can 

15 interact. 

^^fW^Fig. 2, a "personal" computer 34 architecture that can be used for client machine 14 or 
client machine 1 6 is shown in block diagram form. It should be noted that a variety of machine 
architectures can be used to access the Internet 12, i.e. can be used as "network access computers." 
The particular architecture shown for the computer 34 is a typical personal or "PC computer 

20 architecture, such as that used with IBM compatible personal computers. Web server machines can 
also have similar architectures, but are often more powerful computers known as "workstations" that 
operate under some variant of the UNIX® operating system. The Internet service providers 20a are 
likewise often UNIX-based computers or powerful personal computers running Windows NT*. The 
nodes 20 are most commonly routers built by Cisco Systems of San Jose, California. Client 

25 machine 14 or 16 can also take other forms, such as a television including or connected to a 
microrprocessor for Internet access. Force feedback devices used with such client machines can be 
appropriate for the particular embodiment, e.g., a TVremote control used for internet browsing on 
the abovementioned television can include force feedback functionality. 

The personal computer system 34 includes a microprocessor 36 clocked by a system clock 
30 CLK and which is coupled to a high speed or memory bus 38 and to a lower speed or I/O bus 40. 
The system RAM 42 and ROM 44 are typically coupled to the high speed memory bus, while various 
peripherals, such as the video display, hard disk drive, Internet interface (often either a modem or an 
Ethernet connection), and force feedback device, are typically coupled to the slower I/O bus. The 
microprocessor executes programs stored in the various memories (RAM, ROM, hard disk, etc.) of 
35 the personal computer 34 to control, for example, the image display on the video display and the 
forces provided by the force feedback device. The manufacture and use of personal computers, such 
as personal computer 34, are well-known to those skilled in the art. 

In Fig. 3, a client machine 46 in accordance with the present invention includes a personal 
computer system 48 and a force feedback human/computer interface or "force feedback device" 50. 
40 A user 52 can receive visual information 54 and auditory information 56 from the personal computer 
48 and can manipulate the force feedback device 50 as indicated at 58a and 58b to provide input, 
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5 e.g., to command a cursor location on a visual display or other provide other control information. In 
addition, the user 52 can receive force feedback 60 from the force feedback device 50 to represent 
physical "feel" sensations. 

The personal computer system 48 includes the microprocessor 36, the system clock 62, a 
video monitor 64 (which is one type of "visual display"), and an audio device 66. The system clock 
10 62, as explained previously, provides a system clock signal CLK to the microprocessor 36 and to 
other components of the personal computer system 48. The display device 64 and the audio output 
device 66 are typically coupled to the I/O bus 40 (not shown in this figure). 

In this preferred embodiment, the force feedback device 50 preferably includes a local 
microprocessor 68, a local clock 70, optional local memory 71 for the local microprocessor 68, a 
15 sensor, interface 72, sensors 74, a user manipulatable object 76, "other" input interface 78, an 
actuator interface 80, a safety switch 82, and actuators 84 which provide a force F to the object 76, 
and an optional power supply 86 to provide power for the actuator interface 80 and actuator 84. 

The microprocessor 36 of the personal computer system 48 is coupled for communication 
with the local microprocessor 68 of the force feedback device 50. This communication coupling qan 

20 be through a serial port coupling 88 to the personal computer system, or through a game port 
coupling 90 to the personal computer system. Virtually all personal computer systems built to the 
IBM PC/AT standards will include a serial port and a game port. As noted, the serial port will permit 
two-way communication between microprocessor 36 and microprocessor 38, and thus is preferable 
over the game port coupling which only permits one-way communication from the local processor 68 

25 to the microprocessor 36. In consequence, a serial port connection between the personal computer 
system 48 and the force feedback device 50 will permit force feedback commands to be sent from the 
microprocessor 36 to the local microprocessor 68, while a game port connection alone will not be 
able to provide this function. However, some simpler forms of "reflex" type force feedback can still 
be provided by the force feedback device 50 under the control of the local microprocessor 68 even if 

30 only a game port interface is used. It should also be noted that the microprocessor 36 and a local 
microprocessor 68 can communicate over both the serial port connection and the game port 
connection to provide a greater communication bandwidth. A preferred serial port is the Universal 
Serial Bus (USB) of a personal computer, although an RS-232 serial bus, or other serial busses, a 
parallel bus, an ethernet bus, or other types of interfaces or communication links can also be used. 

35 In use, the user 52 of the client machine 46 grasps the object 76 of the force feedback device 

50 and manipulates (i.e. exerts a force to move or attempt to move) the object to cause a "pointer" 
icon to move in the image displayed by the display device 64. This pointer icon typically takes the 
form of a small arrow, a pointing hand, or the like. The sensor 75 senses the movement of the object 
76 and communicates the movement to the local microprocessor 68 through the sensor interface 72. 

40 The local microprocessor 68 then communicates through serial port 88, game port 90, or both to the 
microprocessor 36 to cause the microprocessor 36 to create a corresponding movement of the pointer 
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5 icon on the image displayed upon the visual display 64. In some embodiments, the sensors 74 can 
communicate directly to microprocessor 36 without the use of local microprocessor 68. The user can 
also create other input, such as a "button click" through the other input 78 which are communicated 
to the microprocessor 36 by the local microprocessor 68 or directly, e.g.. using a game port. 

If the pointer icon on the display device 64 is at a position (or time) that correlates to a desired 
10 force feedback to the user 52. the microprocessor 36 sends a force feedback command to the local 
microprocessor 68 over the serial port connection 88. The local microprocessor 68 parses this force 
feedback command and sends signals to the actuator interface 80 which causes the actuator 84 to 
create forces F on object 76, which are experienced by the user 52 as indicated at 60. The safety 
switch 82, sometimes referred to as a "deadman switch", blocks the signal from the actuator interface 
15 80 if, for example, the user 52 is no longer grasping the object 76. In this way, the user 52 can 
interact with the client machine 46 in a visual, auditory, and tactile fashion. 

In Fig. 4a, a force feedback device 50a is provided with a user manipulatable object 76a 
which, in this instance, includes a shaft 90 and a ball-grip (or joystick) 92. The force feedback 
device 50a also includes a pair of linear voice coil actuators ("voice coils") 94 and 96 that can serve 

20 both as sensors and actuators. Alternatively, the voice coils can be used only as actuators,- and 
separate sensors (not shown) can be used. The voice coil 94 is coupled to the shaft 90 of object 76a 
by a first link 98 and a second link 100. Link 98 is coupled to link 100 with a pivot 102. and a link 
100 is coupled to the shaft 90 by a pivot 104. Similarly, voice coil 96 is coupled to the shaft 90 of 
the object 76a by a first link 106 and a second link 108. The first link 106 is coupled to second link 

25 108 by a pivot 1 10, and the link 108 is coupled to the shaft 90 of the object 76a by the pivot 104. 

The link 98 can move in and out of a housing 1 12 as indicated by arrow 1 14, and link 106 
can move in and out of a housing 1 16 of voice coil 96 as indicated by the arrow 118. The pivots 
102, 104, and 1 10 allow the object 76a to move within the constraints of an x-y plane, but does not 
permit movement in a z direction orthogonal to the x-y plane. Therefore, the force feedback device is 
30 a two degree (2D) of freedom device. That is, the user manipulatable object 76a can move with a 
first degree of freedom in a x direction, and in a second degree of freedom in the y direction. A 2D 
force feedback device 50a is considered preferable in the present invention since it correlates well to 
the two-dimensional screen of a monitor of a client machine. 

In Fig. 4b, a voice coil 94 is shown in a cross sectional view taken along line 4b-4b of Fig. 

35 4a. The housing 112 includes a central core 120 and a number of elongated magnets 122. An 
armature 124 includes a hollow, cylindrical member having inner surface 126 which slidingly 
engages the core 120. Wrapped around the armature 124 are coils 128. The coils are electrically 
coupled to actuator and/or sensor interfaces. A plate 130 is attached to the end of the armature 124 
and is coupled to the link 98. The armature 124 and link 98 can move in a linear fashion as indicated 

40 at 114. Other voice coil configurations can also be used, such as differently shaped cores, different 
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5 coil layouts, etc. In addition, other types of actuators can be used instead of or in addition to voice 
coils in force feedback device 50, such as DC motors, brakes, pneumatic actuators, etc. 

In Fig. 5a, an alternative embodiment of a force feedback device 50b is illustrated. The force 
feedback device 50b has many points of similarity with the force feedback device 50a. with like 
reference numerals referring to like elements. The force feedback device 50b includes the user 

10 manipulatable object 76a, the shaft 90, the ball knob 92, and the voice coils 94 and 96. However, 
the links of the force feedback device 50a have been replaced by flexure members. More 
particularly, the links 98 and 100 of force feedback device 50a have been replaced by a rigid 
connector 132 and a flexible member 134 (collectively comprising a "flexure member"), and the links 
106 and 108 of the force feedback device 50a have been replaced by a connector member 136 and a 

15 flexible member 138 (also collectively comprising a flexure member). The connector 1 32 is rigidly is 
attached to the plate 130 at the end of the armature of the voice coil 94 and is rigidly attached to an 
end of the flexible member 134. The other end of the flexible member 134 is attached to a base 140 
which, in turn, is rigidly attached to the shaft 90 of the object 76a. Similarly, the connector 136 is 
attached to a plate of an armature of voice coil 96 at one of its ends, and is attached to the flexible 

20 member 138 at the other of its ends. The remaining end of flexible member 138 is rigidly attached to 
the base 140. 

The flexible members 134 and 138 serve the same functions as the links of the force feedback 
device 50a described previously. As the object 76a is moved back and forth along an x-y plane, the 
flexible member 134 can move in and out of the voice coil housings 94 and 96, respectively, and can 
25 bend to accommodate angular movement with respect to the x and y axis. This permits the 
connectors 132 and 136 to move back and forth within the voice coils 94 and 96. respectively 

In Fig. 5b, an alternative user manipulatable object 76a takes the form of a stylus 142 which 
can engage an aperture 144 in an alternative base 140'. The alternative base 140' can be coupled to 
the flexible members 134 and 138 of the embodiment of Fig. 5a. Alternatively, the tip of stylus 142 
30 can be rigidly or rotatably attached to alternative base 140' with, for example, a ball joint or other 
joint or fastener. 

In Fig. 5c, another alternative base 140" is provided with an enlarged aperture 144* which 
can be engaged by the tip of a finger 146 of the user. The base 140" then becomes the user 
manipulatable object 76c. As before, the base 140" is coupled the flexible members 134 and 138 of 
35 the first feedback device 50b of Fig. 5a. 

The embodiments of figures 5b and 5c illustrate two of a range of equivalent user 
manipulatable objects suitable for the present invention. It should be apparent to those skilled in the 
art that these alternative objects 76b of Fig. 5b and 76c of Fig. 5c can equally well be used with other 
force feedback devices, such as the force feedback device 50a illustrated in Fig. 4a. 
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As noted previously, a preferred embodiment of the present invention provides a user 
manipulatable object that has two degrees of freedom. Other user manipulatable objects having one 
degree of freedom or three or more degrees of freedom are also within the scope of the present 
invention. For example, one embodiment of the present invention provides only one degree of 
freedom. Other force feedback devices of the present invention include mice, joysticks, joypads. a 
steering wheel, and yolks having two or more degrees of freedom. 

In Fig. 6, a conceptual representation of the network system 10 with force feedback includes 
a server machine 1 8, a client machine 14 provided with a force feedback device 24, and one or more 
additional client machines 16, each of which may be provided with additional force feedback devices 
26. As noted in this figure, the server machine is a computer or "processor 4 * running, for example, 
the TCP/IP server software and is which is connected to the Internet. The client machine 14 includes 
a computer or "processor" running Internet browser software and force feedback driver software. 
The processor of the client machine is connected to the Internet and to the force feedback device 24. 
The force feedback device 24 has sensors and actuators so that it can track movement of the user 
manipulatable object, monitor for button presses and/or other ancillary input devices, and provide 
output force feedback sensations. The force feedback device 24 sends object tracking information to 
the client machine, and receives force feedback commands from the client machine 14. -The 
"additional client", such as client machine 16, also includes computers or "processors" running 
Internet browser software and force feedback driver software. The processors of these additional 
clients are also connected to the Internet and are connected to force feedback devices associated with 
that client. 

As noted in Fig. 6, a client machine 14 can send a data request to the server machine 18 and, 
in return, receive an HTML web page file including a special file of the present invention known as 
an "IFF' file. As will be appreciated by those skilled in the art. the server must also have a modified 
configuration file which lets it know that .IFF is a valid MIME type. This modified file would be a 
SRM.CONF or other .CONF file. The client machine 14 then sends force feedback commands to 
the force feedback device 24 and receives tracking and button data from the force feedback device 24. 
Client machine 16 can likewise send a data request to the server machine 18 and receive an HTML 
file with one or more IFF files. The client machine 16 can then interact with the force feedback 
device 26 by sending force feedback commands to the device 26 and by receiving tracking and button 
data from the force feedback device 26. 

In addition to communicating with the server machine, the client machines can communicate 
directly with each other over the Internet using an Internet communication protocol. For example, 
client machine 14 can communicate with client machine 16 through a TCP/IP connection. This is 
accomplished making the URL of the client machine 16 known to the client machine 14, and vice 
versa. In this fashion, direct communication between client machines can be accomplished without 
involving the server machine 18. These connections can send force feedback information and other 
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5 information to the other client machine. For example, a process on the client machine 16 can send 
force feedback information over a TCP/IP Internet connection to the client machine 14. which will 
then generate a force feedback command to the force feedback device 24. When the user reacts to the 
force feedback at force feedback device 24, this information can be sent from client machine 14 to 
client machine 16 to provide force feedback to the user on force feedback device 26. 

10 In Fig. 7a, a flow diagram illustrates an "acquire URL" process 146 running on a client 

machine, such as client machine 14 or client machine 16. This process 146 is preferably 
implemented using a standard Internet browser with a "plug-in" extension which permit the handling 
of force feedback commands. A preferred browser software is Netscape Navigator^ software 
available from Netscape Corporation of Mountain View, California. The plug-in software is a 

1 5 proprietary extension of the web browser software, where this proprietary extension was developed 
by the Applicant of the present application. 

The process 146 begins at 148 and, in a step 150, a connection request is sent to the "host" of 
the desired URL. The host, in this example, is a server machine 18 and the desired URL is the URL 
of the desired web page residing on the server machine 18, the web page including force feedback 

20 commands. Alternatively, the desired web page can reside on another server or resource and -be 
retrieved by server machine 18. In response to the connection request of step 150, the server 
machine 18 sends the HTML file representing the web page over the Internet to be received by the 
client machine. The HTML file includes a number of "components" which are typically commands, 
command fragments, instructions, and data which permit the display of the web page and other web 

25 browser functionality. In a step 154, and an HTML component is obtained. If this component is the 
end of file ("eoO, a step 156 detects that fact and the process is completed at 158. Otherwise, the 
HTML component is parsed and interpreted at a step 160 and process control is returned at step 154. 

It should be noted that most web browser software will start parsing and interpreting (i.e. 
processing) the HTML components even before the entire HTML file is received at the client 
30 machine. Alternatively, the entire HTML file can be received before the processing begins. 

In Fig. 7b, an example of an HTML web page 32, sent from a web server machine 18 to a 
client machine (such as client machine 14 or 16) over the Internet 12, is shown. The HTML file 32 
includes a number of "components" which are parsed and interpreted as previously described. An 
HTML file begins with a <IITML>command 162 to indicate the start of the HTML file, and a 

35 <BODY> command 164 to indicate that the body of the HTML file is beginning. Then, an arbitrary 
number of HTML commands 166 are provided to. for example, display images of the web page on 
the video display of the client machine. A <CENTER> command 168 will cause a centering of 
following objects with respect to the browser window on the video display of the client machine. 
Next, an <EMBED ...> command 170 of the present invention defines a force button object that will 

40 be displayed on the client machine. Since the <CENTER> command 168 was given just prior to the 
<EMBED ...> command, this "force button" will be centered in the displayed browser window. 
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5 Other force objects besides button objects can also be defined and displayed, such as links, text, 
sliders, game objects (balls, paddles, etc.), avatars, windows, icons, menu bars, drop-down menus, 
or other objects. 

In a first line 172 of the <EMBED ...> command, the force button object is defined by a 
"IFF* extension file, namely "FORCEBUTTON.IFF." Next, in a line 174, the size of the button is 

10 indicated to be 100 pixels by 100 pixels. In a line 176, the initial state of the button is indicated to be 
"up" (i.e., unselected), and a line 178 defines the force effect to be "vibration." A number of 
parameters 180 defining the character and nature of the vibration are also provided (start time, length, 
frequency, magnitude, etc.). In a line 182, the "trigger" for the force effect is given by the function 
"MOUSEWTTHIN" with its associated parameters, and by the function "BUTTONSTATE." The 

15 function MOUSEWITHIN determines whether the pointer icon, the position of which is controlled 
by the force feedback device, is within the specified boundaries defining a region of the force button. 
This region can be specified by the parameters and, for example, can be defined as the exact 
displayed area of the button, or can be defined as a sub-region within the button that is smaller than 
the displayed size of the button. The function BUTTONSTATE determines whether a button or 

20 switch of the force feedback device is in the desired state to trigger the force object event (e.g., a 
button event in this example). In a line 184, the icon representing the force button is specified as 
"LOUIS.GIF " and the text associated with the button is defined as "Hi, I'm Louis" in a line 186. 
The font of the text is given as "Helvetica" in a line 188. Other force effects, triggers and parameters 
can also be associated with the force object. For example, a force (such as a vibration) can be 

25 triggered if the pointing icon is moved a predetermined velocity or within a predefined range of 
velocities within the force object. Or, a trajectory of the pointing icon on a force object can trigger a 
force, like a circle gesture. 

The <EMBED ...> command is an existing functionality of HTML. It essentially embeds 
function calls which are handled by the web browser. If the suffix of the specified file is a known, 

30 standard suffix type, the call is executed directly by the web browser. If, however, the suffix (.IFF 
in this instance) is not a standard feature of the web browser, the browser will first look for a "plug- 
in" to implement this feature and, if a suitable plug-in is not found, it will look for application 
programs implementing this feature. In the preferred embodiment of the present invention, a plug-in 
including a reference to a Dynamically Linked Library (DLL) is provided to give functionality to the 

35 .IFF suffix. The DLL can be provided local to the client machine or on another linked resource. 

With continuing reference to Fig. 7b, the centering command is terminated at line 190 with 
the </CENTER> command. Additional HTML commands 192 can then be provided, and the body 
of the HTML file is terminated by the </BODY> command 194. The end of the HTML file is 
indicated at 196 with the </HTML> command, i.e. this command 196 is the "eof ' command of the 
40 HTML file 32. 
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5 The present invention also provides for programmability of the embedded force feedback 

object. An example of this programmability is shown at 198. This optional programmable command 
can be inserted into the EMBED command 1 70 and can include, for example, an iterative loop. In 
line 200, a "FOR" command initializes a counter i to 0, indicates that the counter 1 is incremented by 
one per each pass through the loop, and it indicates that the loop should be completed five times, i.e. 

10 while i<5. The body of the loop includes a command line 202 which indicates that a force feedback 
"vibrate" with associated parameters should be evoked, and a line 204 indicates that a 5 second wait 
should be provided after the vibration has occurred. This step will repeat five times, i.e. the 
command 198 will cause five vibration sequences separated by four 5 second pauses, and followed 
by a final 5 second pause. By providing programmability to the force feedback object, force 

15 feedback effects based upon past events and upon a complex interaction of factors can be provided. 

In Fig. 8, the "Parse And Interpret HTML Component" or simply "Process HTML 
Component" step 160 of Fig. 7a is illustrated in greater detail. In Fig. 8, process 160 begins at 206 
and, in a step 208, it is determined whether there is an embedded "tag" for a force object, e.g. a tag 
having an .IFF reference. An example of the embedded tag is shown at the EMBED command 170 
20 of Fig. 7b. If there is such a tag, step 210 uses the plug-in software of the present invention to 
interpret the .IFF file, and the process is completed at 212. Otherwise, another type of HTML 
command has been encountered, and the standard web browser parser and interpreter processes this 
HTML component in a step 214, after which the process is completed at 212. 

In Fig. 9, the step 210 "Plug-In Interprets .IFF File" of Fig. 8 is described in greater detail. 

25 Process 210 begins at 216, and in a step 218, a "framework" is created for the force object. The 
framework provides a particular set of generic features to implement the specified force object, and 
preferably includes no specific parameters or functions for the force object. Next, in a step 220, the 
name/value pairs are parsed and, in a step 222, the force object is built upon this framework based 
upon the name/value pairs. A name/value pair includes the name of a component and its associated 

30 parameters. For example, one name might be "BUTTONSTATE" and its value (or parameter) might 
be "UP 1 (or "UNSELECTED"). The process 210 is completed at 224. 

In Fig. 9a, an image 226 to be displayed on a screen of a video monitor or other visual 
display is illustrated. More specifically, image 226 can be generated by the popular Netscape 
Navigator® web browser. The image 226 includes a window 228 including a header portion 230 and 

35 a body portion 232. The header portion 230 includes a number of navigation buttons 234 and special 
purpose buttons 236 for purposes well-known to those familiar with the Netscape Navigator web 
browser. In addition, the header portion 230 displays the URL of the currently displayed web page 
at 238. In this instance, the URL is "http://www.imrnerse.com/demo." The images displayed within 
the body portion 232 of the window 228 are created by the aforementioned processing of the HTML 

40 file by the web browser. 
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5 The area within the body portion 232 has been provided with a number of regions and 

buttons to illustrate some of the concepts of the present invention. The force feedback device 
controls the position of a pointer icon 240 which can be caused to interact with the various regions 
and buttons. As an example, when the force feedback device is manipulated by the user to cause the 
pointer icon 240 to move within a "texture" region 242, force feedback commands can be created for 

10 the force feedback device to provide a desired "texture" to the force feedback device. For example, 
the texture can feel "rough" to the user by causing the force feedback device to place forces on the 
user manipulatable object that emulate a rough or bumpy surface. In a region 244, a "viscosity" 
force feedback can be provided. With this form of force feedback, as the pointer icon is moved 
through field 244, a viscous "drag" force is emulated on the user manipulatable object. In a region 

15 246, inertial forces can be felt. Therefore, a pointer icon being moved through an "inertia" region 
would require relatively little or no force to move in a straight line, but would require greater forces 
to accelerate in a new direction or to be stopped. The inertial force sensations can be applied to the 
user manipulatable object and felt by the user. In a "keep out" region 248, the pointer image is 
prevented from entering the region. This is accomplished by creating a repulsive force on the user 

20 manipulatable object using a force feedback command to the force feedback device which prevents or 
inhibits the user from moving the user manipulatable object in a direction of the region 248 when the 
pointer icon 240 contacts the periphery of the region 248. In contrast, a "snap-in" region 250 will 
pull a pointer icon 240 to a center 252 whenever the pointer icon engages the periphery of the snap-in 
region 250 and apply a corresponding attractive force on the user manipulatable object. A "spring" 

25 region 243 emulates a spring function such that a pointer icon moving into the spring region 
"compresses" a spring, which exerts a spring force on the user manipulatable object which opposes 
the movement of the pointer icon. A region 256 is a "Force To Left" region where the pointer icon 
within the region 256 is forced to the left side of the region and the user manipulatable object is 
forced in a corresponding direction as if influenced by some invisible magnetic force or gravitational 

30 force. A region 258 illustrates that regions can be of any size or shape and that within a region 
different force effects can be developed. In this example, within region 258 there is a texture core 
260 surrounded by a vibration ring 262. Therefore, as the pointer icon 240 moves into the region 
258, the user first experiences vibration from the ring 262, and then experiences a texture as the 
pointer icon moves within the core 260. 

35 The exemplary force feedback web page of Fig. 9a is also provided with several force 

feedback buttons. In a first button 264, the placement of the pointer icon 240 over the button and the 
pressing of a button (i.e., a switch) on the force feedback device to create a "button click", "button 
down", or simply a "button event" input, will then cause a "buzz" command to be sent to the force 
feedback device. The buzz command would, for example, cause a vibration force on the user 

40 manipulatable object. Similarly, the selection of the "jolt" button 266 will cause a jolting force (e.g., 
a short-duration pulse of force) to be provided at the force feedback device, and the pressing of the 
"detent" button 268 will cause a "detent" to be created for the force feedback device. By "detent" it is 
meant that the user manipulatable object will be controlled by the force feedback actuators such that it 
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feels as if a mechanical-type detent exists at the position that the user manipulatable object was in 
when the detent button 268 was activated. 



In Fig. 10, a process 270 of the plug-in software of the present invention is illustrated. The 
process 270 begins at 272 and, in a step 274, the position and button state of the force feedback 
device is monitored. Next, in a step 276, a force feedback command is created in response to the 
10 detected position and state. Finally, a command is sent to the Dynamically Linked Library (DLL) to 
place a force feedback command on the interface which can be parsed and interpreted by the force 
feedback device. The process is then completed as indicated at 280. 

It should be noted that the force feedback driver (browser plug-in or DLL) can have the 
ability to interact with JAVA code. In this embodiment, the plug-in reads and executes JAVA 
15 commands using the browser's run-time JAVA interpreter. JAVA can optionally be used to make 
"aplets" which perform dynamic models, such as creating complex force feedback sensations. 

It should also be noted that the force feedback device itself can have a JAVA interpreting chip 
on board, permitting the plug-in driver to download JAVA code to the force feedback device to be 
executed on the device. JAVA and JAVA interpreting chips are available under license from SUN 
20 Microcomputers of Mountain View, California. 

Furthermore, the force feedback driver (browser plug-in or DLL) can have the ability to 
interact with instructions provided in other languages besides HTML. For example, virtual reality 3- 
D graphical environments are increasingly being created and implemented over the World Wide Web 
and Internet using languages such as the Virtual Reality Modeling Language (VRML) and software 

25 such as Active X available from Microsoft Corporation. In these 3-D graphical environments, users 
may interact with programmed 3-D objects and constructs using client computer 14 or 16. and may 
also interact with 3-D graphical representations (or "avatars") controlled by other users over the 
World Wide Web/Intemet from other client computers. Force feedback commands and parameters 
can be provided in the instructions or files of these other protocols and languages and received by a 

30 client computer system in an equivalent manner to that described above so that force feedback can be 
experienced in simulated 3-D space. For example, embedded force feedback routines can be 
included in the VRML data for a virtual environment so that when the user moves into a virtual wall, 
an obstruction force is generated on the user-manipulatable object. Or, when the user carries a virtual 
object in a controlled virtual glove, the user might feel a simulated weight of the virtual object on the 

35 user manipulatable object. In such an embodiment, the force feedback device preferably provides the 
user with three or more degrees of freedom of movement so that input in three dimensions can be 
provided to the client computer. 

While this invention has been described in terms of several preferred embodiments, there are 
alterations, permutations, and equivalents which fall within the scope of this invention. It should 
40 also be noted that there are may alternative ways of implementing both the process and apparatus of 
the present invention. It is therefore intended, jlgit the following appended claims be interpreted as 
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including all such alterations, permutations, and equivalents as fall within the true spirit and scope of 
the present invention. 
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5 CLAIMS 

1 . A method for providing force feedback over a network comprising: 

establishing a connection between a server machine and a client machine over a network, said 
client machine including a visual display and a human/computer interface device having force 
10 feedback; 

receiving from said server machine over said network screen display information and force 
feedback information related to said screen display information; 

displaying on said visual display of said client machine a screen image based upon said 
screen display information; 

15 providing a pointer event with said human/computer interface device with respect to said 

screen image; 

computing a force feedback signal based upon said pointer event; and 

providing a force feedback with said human/computer interface device based upon said force- 
feedback signal that is correlated to said screen image on said visual display. 

20 

2. A method for providing force feedback over a network as recited in claim 1 wherein 
said server machine is a web server, said client machine is a network access computer. 

3. A method for providing force feedback over a network as recited in claim 2 wherein 
25 said web server and said network access computer communicate over said network using TCP/IP 

protocols. 

4. A method for providing force feedback over a network as recited in claim 2 wherein 
said pointer event is at least one of a pointer icon movement on said visual display with respect to 

30 said screen image and a button click of said human/computer interface device. 

5 . A method for providing force feedback over a network as recited in claim 3 wheTtin 
said human/computer interface device has at least two degrees of freedom. 
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5 6. A method for providing force feedback over a network as recited in claim 5 wherein 

said human/computer interface device includes a voice coil actuator for providing at least a portion of 
said force feedback. 

7 A method for providing force feedback over a network as recited in claim 5 wherein 
10 said human/computer interface has exactly two degrees of freedom. 

8. A method for providing force feedback over a network as recited in claim 5 wherein 
said human/computer interface is provided with a local microprocessor which communicates with 
said network access computer, said human/computer interface including sensors and actuators 

15 coupled to said local microprocessor, and wherein said force-feedback signal is a force feedback 
command, said method further comprising: 

parsing said force feedback command on said local microprocessor to control said actuators 
of said human/computer interface in a control loop with said sensors of said human/computer 
interface. 

20 

9. A method for providing force feedback over a network as recited in claim 5 further 
comprising: 

developing on said server machine an HTML file including said screen display information 
and said force-feedback information. 

25 

10. A method for providing force feedback over a network as recited in claim 9 wherein 
said HTML file includes an embedded reference that can be used to create a force feedback object on 
said client machine, and further comprising: 

parsing on said client machine said HTML file and creating on said client machine said force 
30 feedback object. 

. ■ 11. A method for providing force feedback over a network as recited in claim. 5 further 
comprising: 

developing on said server machine an VRML file including said screen display information 
35 and said force-feedback information. 
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5 12. . A method for providing force feedback over a network as recited in claim 10 wherein 

said computing a force feedback signal includes: 

monitoring for a pointer event from said human/computer interface; and 

using said force feedback object and said pointer event to provide said force feedback signal. 



10 13. A method for providing force feedback over a network as recited in claim I . wherein 

said client machine is a first client machine, and further comprising a second client machine coupled 
to said network, said second client machine including a visual display and a human/computer 
interface capable of providing a pointer event, said method further comprising: 

establishing a connection between said first client machine and said second client machine 
15 over said network; 

receiving at said first client machine second client information from said second client 
machine; 

computing a force feedback signal based upon said second client information; and 

providing a force feedback with said human/computer interface device based upon said force 
20 feedback signal. 



14. A method for providing force feedback over a network as recited in claim 13 wherein 
said server machine is a web server, said first client machine is a network access computer, said 
second client machine is a network access computer, and said web server, said first network access 
25 ComputerLand said second network access computer can communicate with said network using 
TCP/IP protocols. 



15. A method for providing force feedback over a network as recited in claim 1 4 wherein 
said pointer event is at least one of a pointer icon movement on said visual display of said first 
30 network access computer with respect to said screen image and a button click of a human/computer 
interface device of at least one of said first network access computer and said second network access 
computer. 



16. A method for providing force feedback over a network as recited in claim 1 5 wherein 
35 said human/computer interface devices of said, first network access computer and said second 
network access computer each have at least two degrees of freedom. 
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17. A method for providing force feedback over a network as recited in claim 16 wherein 
said human/computer interface devices each have exactly two degrees of freedom. 

18. A method for providing force feedback over a network as recited in claim 17 wherein 
10 said human/computer interface devices each include a voice coil actuator for providing at least a 

portion of said force feedback. 



19. A method for providing force feedback over a network comprising: 

15 establishing a connection between a first computer and a second computer over a network, 

where said first computer includes a computer input device providing a first computer input, and 
wherein said second computer includes a visual display and a human/computer interface device 
providing force -feedback in response to a force feedback signal and providing a second computer 
input; 

20 displaying on said monitor of said second computer a screen image that is associated with 

stored force feedback information; 

receiving from said first computer over said network a first computer input associated with 
said screen image on said second computer; 

determining said force feedback signal based upon said stored force feedback information, 
25 and at least one of said first computer input and said second computer input; and 

providing a force feedback with said human/computer interface device in response to said 
force feedback signal that is correlated to said screen image on said second computer. 

20. A method for providing force feedback over a network as recited in claim 1 9 wherein 
30 said first computer and said second computer communicate over said network using TCP/IP 

protocols. 

21. A method for providing force feedback over a network as recited in claim 19 wherein 
establishing a connection includes the exchange of URLs between said first computer and said 

35 second computer. 
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5 22.. A method for providing force feedback over a network as recited in claim 19 wherein 

said first computer input and said second computer input comprise a pointer event including at least 
one of a pointer movement and a button click. 

23. A method for providing force feedback over a network supporting TCP/IP protocols 
10 comprising: 

sending from a client computer over a network supporting TCP/IP protocols a connection 
request to a web server connected to said network that is hosting a desired URL; 

receiving and parsing an HTML file at said client computer that was sent from said web 
server in response to said connection request, wherein said parsing includes parsing an embedded 
15 force object reference having associated parameters and building a force object as specified by said 
force object reference and said associated parameters; 

developing a force feedback signal with said force object; and 

providing force feedback to a human/computer interface device coupled to said client 
computer in response to said force feedback signal. 

20 

24. A method for providing force feedback over a network supporting TCP/IP protocols 
as recited in claim 23 wherein said HTML file comprises a web page of said web server. 

25. A method for providing force feedback over a network supporting TCP/IP protocols 
25 as recited in claim 24 wherein a network browser is provided on said client computer to perform said 

parsing and developing said force feedback signal. 

26. A method for providing force feedback over a network supporting TCP/IP protocols 
as recited in claim 25 wherein said network browser is provided with a plug-in to aid in the parsing 

30 of said embedded force object reference, building said force object, and developing said force 
feedback signal. 

27. A method for providing force feedback over a network supporting TCP/IP protocols 
as recited in claim 26 wherein said human/computer interface device is provided with a local 

35 microprocessor that communicates with said client computer, and wherein said force feedback signal 
is a force feedback command sent from said client computer to said human/computer interface device. 
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28. A method for providing force feedback over a network supporting TCP/IP protocols 
as recited in claim 27 wherein said local microprocessor can interpret JAVA instructions sent from 
said client computer. 



10 29. A method for providing force feedback over a network supporting TCP/IP protocols 

as recited in claim 27 wherein said human/computer interface device includes actuators and sensors, 
and further comprising: 

parsing on said local microprocessor said force feedback command to control said actuators 
in a feedback loop with said sensors. 

15 

30. A networked force feedback system comprising: 
a network; 

a first computer coupled to said network; and 

a second computer coupled to said network, said second computer including a visual display 
20 and a human/computer interface device capable of providing a second computer input and capable of 
providing force feedback in response to a force feedback signal provided by said second computer, 
said second computer developing an image on said visual display that is associated with stored 
feedback information, wherein said second computer produces said force feedback signal in response 
to at least one of information derived from said first computer and of said second computer input. 

25 

31. A networked force feedback system as recited in claim 30 wherein said first computer 
is a server computer, said second computer is a network access computer, and wherein said 
information derived from said first computer includes screen display information and force feedback 
information related to said s6reen display information. 

30 

32. A networked force feedback system as recited in claim 31 wherein said information 
derived from said server computer is sent from the server computer to the network access computer 
in the form of an HTML file. 



35 33. A networked force feedback system as recited in claim 31 wherein said information 

derived from said server computer is sent from the server computer to the network access computer 
in the form of a VRML file. 
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5 

34. A networked force feedback system as recited in claim 30 wherein both said first 
computer and said second computer arc network access computers which communicate over said 
network using TCP/IP protocols. 

10 35. A networked force feedback system as recited in claim 34 wherein said information 

derived from said first computer includes information that affects the force feedback signal. 

36. A networked force feedback system as recited in claim 35 wherein said information 
derived from said first computer and said second computer input both comprise at least one of a 

1 5 pointer icon position input and a button click input. 

37. A networked force feedback system as recited in claim 30 wherein said 
human/computer interface device coupled to said second computer includes a local microprocessor 
that communicates with said second computer, a plurality of actuators for providing a force feedback, 

20 and a plurality of sensors for sensing positions of said human/computer interface device. . 

38. A networked force feedback system as recited in claim 37 wherein said 
human/computer interface device coupled to said second computer includes a base for receiving input 
from said user, said base being movable in two linear degrees of freedom. 

25 

39. A networked force feedback system as recited in claim 38 wherein said base is 
receptive to a finger of said user for manipulating said base in said two linear degrees of freedom. 

40. A networked force feedback system as recited in claim 38 wherein said base is 
30 receptive to a stylus for manipulating said base in said two linear degrees of freedom. 

41. A networked force feedback system as recited in claim 37 wherein said force feedback 
signal includes a force feedback command that can be parsed by said local microprocessor such that 
said microprocessor can control said actuators in response to said force feedback command in a 

35 control loop with said sensors. 
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5 42. A networked force feedback system comprising: 

network means; 

first computer means coupled to said network means; and 

second computer means coupled to said network means, said second computer means 
including visual display and human/computer interface means, said second computer means further 

10 including means for displaying an image on said visual display that is associated with stored force 
feedback information of said second computer means, said second computer means further including 
means for analyzing information sent to said second computer means from said first computer means 
over said network means, said second computer means further including means for developing a 
force feedback at said human/computer interface means using said stored force feedback information 

15 in response to at least one of said information sent by said first computer means and a second 
computer input providing by said human/computer interface means. 

43. A networked force feedback system as recited in claim 42 wherein said first computer 
means is a server computer means, said second computer means is a network access computer 

20 means, and wherein said information derived from said first computer means includes screen display 
information and force feedback information related to said screen display information. 

44. A networked force feedback system as recited in claim 43 wherein said information 
derived from said server computer means is sent from the server computer means to the network 

25 access computer means in the form of an HTML file. 

45. A networked force feedback system as recited in claim 42 wherein both said first 
computer means and said second computer means are network access computer means which 
communicate over said network using TCP/IP protocols. 

30 

46. A networked force feedback system as recited in claim 45 wherein said information 
derived from said first computer means includes information that affects said force feedback signal. 

47. A networked force feedback system as recited in claim 46 wherein said information 
35 derived from said first computer means and said second computer means input both comprise at least 

one of a pointer icon position input and a button click input. 
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48. A networked force feedback system as recited in claim 42 wherein said 
human/computer means interface device coupled to said second computer means includes a local 
microprocessor means that communicates with said second computer means, a plurality of actuator 
means for providing a force feedback, and a plurality of sensor means for sensing positions of said 
human/computer means interface device. 

49. A networked force feedback system as recited in claim 48 wherein said force feedback 
signal includes a force feedback command that can be parsed by said local microprocessor means 
such that said microprocessor means can control said actuator means in response to said force 
feedback command in a control loop with said sensor means. 



-28- 



WO 98/06024 



PCT/US97/12848 




SUBSTITUTE SHEET (RULE 26) 



WO 98/06024 



PCT/US97/12848 




SUBSTITUTE SHEET (RULE 26) 



WO 98/06024 PCT7US97/12848 



3/10 



PERSONAL COMPUTER SYSTEM 



■48 



,-46 



62 



CLK 



SYSTEM 




AUDIO OUT- 


CLOCK 






PUT DEVICE 



r 



36 



MICRO- 
PROCESSOR 



66 



DISPLAY 
DEVICE 



SERIAL 
PORT 

88 H 



! GAME 
PORT 



FORCE FEEDBACK 
DEVICE 



70 



L 



68 





LOCAL 


«« — 


SENSOR 




MICRO- 




INTER- 




PROCESSOR 




FACE 



CLOCK 



LOCAL 
MEMORY 



71 



78 



OTHER 
INPUT 



80 




72 



ACTUATOR 
INTERFACE 



— 



SAFETY I 
S WITCH 



MANIP- 
ULATE] 

58tP~ 



ACTUATORS 



84 



POWER 
SUPPLY 



-86 




ure 3 



SUBSTITUTE SHEET (RULE 26) 



WO 98/06024 



4/10 



PCT/US97/12S48 




figure 46 



SUBSTITUTE SHEET (RULE 26) 





SUBSTITUTE SHEET (RULE 26) 



WO 98/06024 



PCT/US97/12848 



6/10 



18 



14 



24 



SERVER MACHINE 



PROCESSOR RUNNING THE 
HTTP SERVER SOFTWARE, 
CONNECTED TO INTERNET 



DATA 
REQUEST 



10 



HTML FILE 
IFF FILE 



CLIENT MACHINE 



PROCESSOR RUNNING 
INTERNET BROWSER SOFT- 
WARE AND FORCE FEEDBACK 

DRIVER SOFTWARE. THIS 
PROCESSOR IS CONNECTED 
TO INTERNET AND CONNEC- 
TED TO FORCE FEEDBACK 
DEVICE 



16, 



TCP/IP 
CONNEC- 
TION 



.Y. .1 

ADDITIONAL CLIENTS 



TRACKING AND BUTTON DATA 
FORCE FEEDBACK COMMANDS 

26 



FORCE FEEDBACK DEVICE 



THIS DEVICE HAS SENSORS 
AND ACTUATORS SO IT CAN 
TRACK A USER'S MOTION, 
MONITOR BUTTON PRESSES. 
AND PROVIDE OUTPUT FORCE 
FEEDBACK SENSATIONS. IT 
WILL SEND TRACKING AND 
SENSOR INFORMATION TO 
CLIENT. IT WILL RECEIVE 
FORCE FEEDBACK COMMANDS 
FROM CLIENT. 



ADDITIONAL FORCE 
FEEDBACK DEVICES 



figure 6 



SUBSTITUTE SHEET (RULE 26) 



WO 98/06024 



PCT/US97/12848 



7/10 



figure 7a 



146 



158 




DONE 




(^STAR^-^ 



SEND 
CONNECTION 
REQUEST TO 
HOST OF URL 



148 



150 



152, 



I 



RECEIVE 
HTML FILE 



154 



OBTAIN HTML 
COMPONENT 



YES 




160^ 



PARSE AND 
INTERPRET 

HTML 
COMPONENT 



OBTAIN HTML 
COMPONENT 



PARSE AND 
INTERPRET 



-214 



212 




-210 



160 



figure. 8 



SUBSTITUTE SHEET (RULE 26) 



WO 98/06024 



PCT/US97/12848 



8/10 



32 



170- 



<HTML>~ t62 
<BODY>~ 164 




f 166 
<CENTER>--J68 

<EMBED "FORCEBUTTON.IFF;--f72 
SIZE = 100,100—774 
INITSTATE = UP— 176 f 180 
FORCEEFFECT = VIBRATION (PAR AMS)—7 78 
TRIGGER = MOUSEWITHIN(PARAMS) + 

BUTTON STATE-si tfl2 
ICON = LOUIS.GIF-^f 54 
TEXT = "HI, I'M LOUIS"^r$6 
FONT = HELVETICA-~r55 

</CENTER>-M90 



-192 



</BODY>— 794 
</HTML>— 196 




204 



figure 76 



SUBSTITUTE SHEET (RULE 26) 



WO 98/06024 PCT/US97/12848 



(^STAR^^' 



CREATE 
FRAMEWORK FOR 
FORCE OBJECT 



9/10 



216 



-218 



PARSE 
NAME/VALUE ' 
PAIRS 



-220 



BUILD FORCE 
OBJECT BASED ON 
NAME/VALUE PAIRS 



( ^^y 224 



.222 



270 



figure 10 



210 



figure 9 



MONITOR 
POSITION AND 
BUTTON STATE 



-274 



CREATE FORCE 
FEEDBACK COMMAND 

IN RESPONSE TO 
POSITION AND STATE 



-276 



SEND COMMAND TO 
DLL TO PUT MEDIUM 
LEVEL COMMAND ON 
INTERFACE 



-278 




SUBSTITUTE SHEET (RULE 26) 



WO 98/06024 



PCIYUS97/12848 



10/10 



£ 



234 



7k 




■234 

h 



r 



226 



Location: |http://www.immcrse.cofn/demo 



i 1 i ; i r— — i | 1 | 1 I 7 ^238 




L 2 30 



240 



^236 J 



TEXTURE 




^242 



248 



£ 



244 



VISCOSITY 



£ 



246 



TEXTURE 



r 



250 



-228 



252 



SNAP i 
IN » 



£ 



254 



SPRING 



FORCE TO LEFT 



t 



260 



256 




■258 



262 



266 



268 



JOLT 



DETENT 



232 



figure 9a 



SUBSTITUTE SHEET (RULE 26) 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US97/12S48 



A. CLASSIFICATION OF SUBJECT MATTER 

1PC<6) :G06F 3/00, 15/16. 17/30 

US CL : 395/200.8. 200.33, 200.47. 200.54 
According lo Intcnutiontl Patent Clarification (IPC) or to both nauorul cUMification and IPC 



FIELDS SEARCHED 



Minimum documentation searched (cUiaification system followed by cUsiification symboU) 
U.S. : 395/200.8.200.33.200.47.200.54 



Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched 



Electronic data base consulted during the international search (name of data base and. where practicable, search terms used) 
APS. DIALOG 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 


Citation of document, with indication, where appropriate, of the relevant passages 


Relevant to claim No. 


A 


US 5,186,629 A (ROHEN) 16 February 1993, cols 2-3 


1-49 


A 


US 5,184,319 A (KRAMER) 02 February 1993, columns 2-6 


1-49 


Y 


US 5,436,638 A (BOLAS et al.) 25 July 1995, columns 5-7 


1-49 


Y 


US 5,459,382 A (JACOBUS et al.) 17 October 1995, columns 2-3 


1-49 


Y 


US 5,386,507 A (TEIG et al.) 31 January 1995, col. 26 line 35 et 
seq. 


M9 



| I Further document* are listed in the continuation of Box C. Q See patent famdy annex. 



dociaini oofm** the |«ml mm* of the art which a not coniidsred 
to be of particular relevance 

«*f her document published on or after die tnterneoonel fU*i| date 



later document published after the sHeraeOooal fitine, < 
bete and sot in conflict with the application but ailed 
the pnocipte or theory underfyn« (he mvanbon 



doubt* on arawiry cwiai(i) or which i 
m of another cttatien or othe 



aer 
the 

reason (aa »i 



document refemnf to en oral , 
means 

or to the intemouooal filmi dale but later i 



document of particular relevance; the cleaned art 
considered novel or cannot be comidered to mvolv 
when the dociancftt m taken elon* 

document of particular relevance; the cleaned invention < 
considered to mvotvc an mvcntive step nbc 
combated with one or more other such doevoeen 
beasg obvious to * person skilled in the art 

■ enter of the came patent family 



Date of the actual completion of the international search 
30 SEPTEMBER 1997 



Date of mailing of the international search report 

Al DEC 7997 




Name and mailing address of the ISA/US 
Commauioner of Paicntt and Trademarks 
BosPCT 

Washington. O.C 20231 
Facsimile No. (703) 305*3230 



Aui 



MET GECK1U 
Telephone No. (703) 305-9676 



Form PCT/1SA/210 (second iheet)(July 1992)* 



